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ABSTRACT 

The Monongalia County (West Virginia) Even Start Program is 
a comprehensive, rural, home-based family literacy program that provides 
families with adult literacy education, parenting education, and early 
childhood education. Although all Even Start programs conduct some home 
visits, the Monongalia County program is unique in that home visits are the 
primary service delivery mechanism. Following an overview of the national 
Even Start program and the Monongalia County program, this report describes 
an evaluation of program impact on the vocabulary development of child 
participants. The sample consisted of 32 children with an average age of 4 
years, 4 months, and included 5 children for whom English was a second 
language (ESL) and 10 children with an active individualized education plan 
(I:3P) . Even Start staff administered the Peabody Picture Vocabulary Test- III 
(PPVT-III) to the children on a pretest-posttest basis during the 1998-99 
program year. Most children had 7-8 months of program participation. From 
pretest to posttest, PPVT-III scores increased significantly for the full 
group and the subgroups. The effect size of the point gain was "moderate” for 
the full group, "large" for the ESL group, and "very large" for the lEP 
group. The PPVT-III average age-equivalent score increased 13 months during 
an average treatment period of 7 months. Therefore, the children in the 
sample nearly doubled the vocabulary growth expected of them. (Contains 20 
references . ) (SV) 
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EXECUTIVE SUMMARY 



The Even Start Family Literacy Program, started in 1989, is a national program that 
addresses the developmental and educational needs of targeted families. First, the literacy and 
educational needs of the adults in the family are identified and addressed. Second, effective 
parenting skills for the adults are taught. Third, the preschool readiness needs of the children in 
the family are addressed with at-home and/or. center-based early childhood education sessions. 
States solicit grant applications from local agencies and make awards on a competitive basis. 



Monongalia County Even Start Program 

The Monongalia County (WV) Even Start Program began in 1993. As required, the 
program consists of three interrelated approaches to address family literacy and prepare children 
for entrance into schooling: adult education, parenting education, and early childhood education. 

The unique feature of the Monongalia County Even Start Program is that it is mainly a 
home-based program. While the law requires that all Even Start programs conduct some home 
visits, in the Monongalia County Program home visits are the primary service delivery 
mechanism. Four family educators work with their assigned families to develop and schedule a 
plan of service. Typically, these biweekly home visits last about one and one half hours each. A 
home visit usually begins with the collection of books and associated toys or materials loaned 
during the prior visit and then moves into a learning activity with the child. Next, while the child 
is occupied, the Family Educator discusses child development issues or parenting concerns with 
the parent. This is followed by some adult education. The home visit concludes with an inquiry 
about other needs the family might have that might be hindering progress in educational or other 
areas and then scheduling the next visit. 



Evaluation Design 

In the fall of 1996, program staff contracted with AEL, Inc. (fomierly Appalachia 
Educational Laboratory) in Charleston, West Virginia, to conduct an independent evaluation of 
the Even Start Program. Overall, AEL’s first evaluation (Meehan, 1997) concluded that the 
Monongalia County Even Start Program was a comprehensive, rural, home-based family literacy 
program that produced many positive impacts on the adults and children studied in the random 
sample. A second evaluation of the program was conducted by AEL staff on a sample of 
families not in the first sample and with different assessment instruments. This second 
evaluation (Meehan, 1998) showed much more impact on the adults in the mathematics areas 
(number operations and problem solving) than in the area of reading comprehension. On the 
parenting instrument, it was concluded that the program had some impact on parents’ internal 



iv 




7 



locus of control and self-esteem but little impact on their parental responsibility and no impact on 
their parenting efficacy. 

With two evaluations completed using two different instruments each to assess 
adult/parent outcomes and just one instrument to assess child outcomes, the Monongalia County 
Even Start staff chose to explore other child outcome measures for use in the 1998-99 program 
year. Staff selected the widely known Peabody Picture Vocabulary Test-Ill (PPVT-III) as a new 
child outcome measure. Staff administered and scored the PPVT-III to a sample of Even Start 
children on a pretest-posttest basis in the 1998-99 year. Even Start staff contracted with AEL 
staff to statistically analyze the PPVT-III scores as the third evaluation in the series. The purpose 
of this evaluation effort was to analyze, display, and discuss the pre- posttest administration of 
the PPVT-III measure in the one year. 

The number of children in the sample was 32. This was less than the number of families 
in the program because not all children completed both pretest and posttest administrations of the 
PPVT-III. There were two subgroups within the sample: five in the English as second language 
(ESL) subgroup and ten in a subgroup that had active individualized education plans (lEP). Data 
analyses included descriptive statistics on treatment months and PPVT-III scores. T-tests for 
dependent means by the matched pairs technique were computed for the full group and the two 
subgroups. PPVT-III age equivalent scores also were analyzed, and effect sizes were computed. 



Findings 

The average age of the 32 children in this study was 51 .66 months or four years and four 
months. The mode and median age in months both were 5 1 .00. 

The treatment period for this evaluation was the 1998-99 Even Start program year. Of the 
32 children with both scores, 15 had eight months of treatment, 10 had seven months, and the 
rest had nine (N=l), six (N=2), five (N=2), or three (N=2). The mean number of treatment 
months was 6.93 with a standard deviation of 1.63. 

The PPVT-III standard scores could range from 90 to 160 points. The pretest mean score 
for the full group was 90.53 with a standard deviation of 14.37. The posttest mean increased to 
99.81 while the standard deviation decreased to 9.38. This gain yielded a t-value of 4.51, which 
was significant at the .0001 level. The effect size for the gain in the vocabulary score was .65, 
which is “moderate” in Cohen’s (1977) set of descriptors. 

For the active lEP subgroup of 10 children, the vocabulary gain was much larger than for 
the full group. The lEP gain was a full 15.50 points. The t-value of 3.56 was slightly less (owing 
to the smaller size of the subgroup), but still it was significant at the .01 level. The effect size 
was 1.13, which is .33 above the start of the “large” category in Cohen’s scheme. Thus, the 



descriptor of “very large” was used to describe the effect size for the active lEP subgroup on the 
vocabulary measure. 

The PPVT-III scores for the ESL subgroup of five increased 14.80 points, a much larger 
increase than that of the full group. The t- value was 3.37 and the probability was .028, 
significant at the .05 level. The ESL subgroup effect size was 1 .05, which is “large” in Cohen’s 
scheme. 

The average age equivalent score of vocabulary for the full group at pretest was 3.71 or 
three years and seven months. At posttest, it was 4.84 or four years and eight months. 



Conclusions 

Several conclusions were drawn from the findings of this evaluation. First, the 9.28 point 
gain on the PPVT-III by the full group was statistically significant and its effect size was 
“moderate.” Thus, the vocabulary gain for the full group was meaningful both statistically and 
practically. Second, the 15.50 point gain on the PPVT-III for the active lEP subgroup was 
statistically significant and its effect size was “very large.” Thus, we conclude with much 
confidence that the vocabulary gains for the active lEP subgroup were meaningful in both the 
statistical and practical sense. Third, the 14.80 point gain for the ESL subgroup was “large.” 
Therefore, it was concluded with much confidence that the PPVT-III gain for the ESL group was 
meaningful in both the statistical and practical sense. 

The PPVT-III average age-equivalent score increased from 3.7 at pretest to 4.8 at posttest, 
or 13 months. The average pretest to posttest treatment period was seven months. This results in 
a net gain of 6 months above the growth in vocabulary expected for normal development. 
Therefore, it was concluded that as a result of being in the Monongalia County Even Start 
Program, the children in the sample nearly doubled the vocabulary growth expected of them. 



Recommendations 

Even Start staff can be certain that their program impacts on the vocabulary gains of 
children in their families. Too, these vocabulary gains are nearly double what would be expected 
in a typical treatment year. Staff should acknowledge and celebrate these student vocabulary 
gains. 



The PPVT-III is a well-known and respected measure. With these convincing findings in 
vocabulary gains by children in the Even Start program, staff may want to continue to administer 
the PPVT-III to all children in the Even Start program in the future. 
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INTRODUCTION 



This section presents an overview of the National Even Start Program, a description of 
the Monongalia County Even Start Program, the purposes and objectives of this evaluation, and 
an explanation of the audience for this report. 



The National Even Start Program 



The Even Start Family Literacy Program is a national literacy program that identifies and 
addresses the educational and developmental needs of targeted families. These developmental 
needs of eligible Even Start families are addressed three different ways. First, the educational 
and literacy needs of the adults in the family are identified and addressed. Second, effective 
parenting skills for the adults are taught. Third, the developmental and preschool readiness needs 
of the children in the family are addressed with at-home and/or center-based early childhood 
education sessions. 

The national Even Start program began in 1989. Legislative authority for the program is 
in the Hawkins-Stafford Elementary and Secondary School Improvement Amendments of 1988 
to the Elementary and Secondary Education Act of 1965. Specifically, the program is Part B, 
Chapter 1 of Title I in Public Law 100-297 (Tao, Swartz, St. Pien'e, & Tarr, 1997). Two years 
later, the original Even Start legislation was amended with the passage of the National Literacy 
Act (Public Law 102-73). Reauthorization for the Even Start program was provided by the 
Improving America’s Schools Act as Part B of Title I. 



Emphasis on Families 

The purpose of the Even Start program is to “help break the cycle of poverty and 
illiteracy” by working with low-income families to improve their educational opportunities. The 
legislation spells out those educational opportunities as being adult basic education or adult 
literacy, parenting education, and early childhood education. Further, the Even Start legislation 
requires that these three educational areas be integrated “. . . into a unified family literacy 
program” (P.L. 103-382, Sec. 1201, as cited in Tao et al., 1997, p. 2 ). The Even Start program 
represents an innovative combination of adult literacy or adult basic education, parenting 
education, and early childhood education. 

Criteria for eligibility into the Even Start program were established in the legislation. 
Participants had to be (1) a parent or parents who (a) are eligible for participation in an adult 
basic education program under the Adult Education Act, or (b) are within the state’s compulsory 
school attending age ranges; and (2) the child or children, from birth tlirough age seven, of the 
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parent or parents described in (1). The (b) part of the parent eligibility requirement was new in 
1995-96, which expanded the Even Start program to include teen parents for the first time. 
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Program Services 

Even Start program services can be divided into two types: (1) “core” educational 
services and (2) ancillary support services. As mandated in the Even Start legislation, the “core” 
educational services are in three interrelated areas: 

(1) high-quality instructional programs that promote adult literacy 

(2) high-quality instructional programs that empower parents to support the 
educational growth of their children 

(3) developmentally-appropriate early childhood educational services 
and preparation of children for success in regular school programs 
(P.L. 103-761, Sec. 1205) 

The legislation requires that Even Start programs provide these core services to parents and 
children jointly and also provide some home-based services. 

Even Start programs are required to provide support services to participating families. 

The twin support services of child care and transportation for the purposes of enabling parents 
and their children to participate in Even Start programs are named specifically in the law. Local 
programs typically provide a range of other support services to their participating families, 
including health care, vision and hearing screening, employment assistance, mental health 
referrals, cross-agency assistance, nutrition education, referrals for screening or treatment for 
chemical dependency, and public library card attainment assistance. 

Even Start initially began as a demonstration program administered by the U.S. 
Department of Education (USDE). The USDE funded 76 Even Start projects in fiscal year 1989 
for a total of $14.5 million. In 1991, the Even Start program expanded to the level of $49.7 
million. Then, in 1992, 100 new Even Start projects were started, totaling $70 million. And, 
according to the law, since the total budget for the program exceeded $50 million, its 
administration was transferred to the states. 

States receive their Even Start funding based on a formula related to the allocation they 
receive from the federal government for their Title I Program. The states then solicit grant 
applications and make awards on a competitive basis. The law provides a portion of the state’s 
allocation for state-level administration and management. 
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The Monongalia County Even Start Program 



Program Setting Information 

Monongalia County, West Virginia, is located in the north central part of the state, 
bordering the Pennsylvania state line. The population of the county, according to the 1990 U.S. 
Census, was 75,599 in its 417.85 square miles, or 180.92 residents per square mile. The county 
seat is the city of Morgantown, the fifth largest city in West Virginia. Approximately 34% of the 
total county population lives in Morgantown, with the remaining 66% living in the outlying, 
mostly rural and mountainous communities (Even Start Staff, 1995, p. III-3). Morgantown also 
is the home of West Virginia University, the largest university in the state. 

The Monongalia County school system had 10,079 students in the 1996-1997 school year 
in grades K to 12. There were 28 schools in the system and 598 teachers (Quality Education 
Data, 1996, p. 14). In 1995, 3,615 (35%) of the system’s children qualified for free and reduced 
lunches (Even Start Staff, 1995, p. III-3). As in all of West Virginia, the vast majority (93%) of 
the school children are Caucasian. It is reported that 54% of the system’s students are college- 
bound, while the dropout rate of 17.5% is very close to the state’s average of 17.7% in 1995. 

The total number of computers available for instructional use was reported as 939, of all brands 
(Quality Education Data, 1996, p. 14). Each school does have a Local School Improvement 
Council and a Faculty Senate. The predominant organizational arrangement for the 20 
elementary schools is K-5 or K-6, although there is one K-3 school, two K-4 schools, and one 
school with no kindergarten. There are two middle schools (one grades 5-8 and one grades 6-8), 
two middle or junior high schools of grades 7-9, one small grades 7-12 school of 41 5 students, 
two large high schools (one grades 9-12 and one grades 10-12) of over 1,100 students each, and 
one vocational-technical education center. The adult education center is housed in the 
vocational-technical center. The Monongalia County Even Start Program is housed in the 
Dorsey Education Center, a renovated school building that also houses the administrative offices 
for the Title I Program, the Head-Start Program, the Early Head Start Program, and two Head 
Start classrooms. 

Poverty, unemployment, and illiteracy exist in Monongalia County. Even Start staff 
reported 38% of the school district’s students are from low-income families (Even Start Staff, 
1999). The total caseload of families receiving health and/or human services in the county was 
reported as 5,174 in July 1994 (Even Start Staff, 1995, p. III-3). The number of persons 
unemployed in 1995 was reported as 2,580, which placed the county in the top seven in the state 
in this category. Recent information shows 10,589 (25%) of the county’s adults do not possess a 
high school diploma (Even Start Staff, 1999). There are 1 1 Title I schoolwide projects.in the 
district’s elementary schools where 51 to 84% of the students qualify for free and reduced-price 
lunches. 
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Program Description 

The Monongalia County Even Start Program, one of seven in the state, began in 1993. 
The program consists of three interrelated approaches to address family literacy and to prepare 
children for entrance into schooling. These tliree approaches are adult education, parenting 
education, and early childhood education. The Even Start staff explain their goals as follows. 

Adult Education - to provide and strengthen opportunities that will increase 

the literacy and education of parents 

Parenting Education - to help parents gain competence in effective parenting 

practices that will maximize their child’s development and 
assist them in becoming full partners in their child’s 
education 

Early Childhood Education - to assist children in reaching their full potential as learners 

(1995, p. III-l) 

The Even Start program in Monongalia County is designed to provide simultaneous 
education services to participating parents and their children. The program has been designed to 
inform and educate parents on how important their involvement is in their child’s education and 
then to teach parents how to achieve this goal. The Monongalia County Program staff has 
received training in the Parents As Teachers (PAT) program and has adapted it for its parenting 
education curriculum. The PAT is a nationally-validated home-school partnership program 
designed for 0- 5-year-old children. Its purpose is to provide the training and curriculum to 
support parents in giving their children the best possible start in life (Even Start Staff, 1995, p. 
III-l). 



In addition to the PAT curriculum, the Monongalia County Even Start Program has 
developed and implemented its own family literacy model. This locally-developed model 
emphasizes children’s literature in fostering cognitive and educational growth in the family unit. 
A unique component of this model is that the program has developed an extensive lending library 
of children’s books; developmentally-appropriate toys; and other learning materials such as 
crayons, paints, markers, and art paper. These literacy development materials are loaned to Even 
Start families during regular, planned home visits. For example, a young boy might choose a 
book such as Tommy The Truck to read and a Tonka truck might be brought for him to play with 
before the next home visit by the Family Educator (described below). 

The unique feature of the Monongalia County Even Start Program differentiating it from 
most other programs is that it is primarily a home-based program. While the law requires that all 
Even Start programs conduct some home visits, in the Monongalia County Program, home visits 
are the primary service delivery mechanism. The four Family Educators work with their 
assigned families to develop and schedule a plan of service. Most families are visited every other 
week in their homes. Typically, these biweekly visits last about one and a half hours each, but 
vary according to the family’s needs and other factors. During visits, the Family Educator 
provides meaningful instruction to both the child or children present and to the parent. Typically, 

O 
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a home visit begins with the collection of books and toys loaned during the prior visit, and then 
moves into a learning activity that focuses on literacy with the child or children. Next, while the 
children are occupied, the Family Educator discusses child development issues or addresses 
parenting concerns such as discipline (Even Start Staff, 1995, p. III-2). Even Start Staff reported 
making a total of 1,103 home visits in 1997-1998 and circulating 3,388 books (1998, P. III-31). 

During home visits, the Family Educator also completes assessments of other family 
needs. Here, the occasion of being in the home is used to inquire and talk about other needs that 
might be hindering progress in educational or other areas. Examples include vision and hearing 
screening; assistance in securing family benefits; assistance in looking for and obtaining jobs; 
help in receiving treatment for chemical dependency; and help in enrolling and transporting 
children to Head Start, kindergarten, or other school programs. When appropriate, the Family 
Educators will make referrals to other agencies in the county to which the Even Start program is 
linked — many in a formal, collaborative manner. Also, whenever it can be arranged, the Family 
Educators encourage and foster communications across the participating families and sponsor 
events and activities for all parents or subgroups of parents with similar needs. A group meeting 
at the local public library, where parents received their own library card, is one example. 

Figure 1 is a graphic depiction of the Monongalia County Even Start Program with its 
eight essential components. This graphic portrayal of the program is a product of the prior 
evaluation of the Monongalia County Program (Meehan, 1997). 



Program Information and Accomplishments 

As part of its record keeping and reporting functions, the Monongalia County Even Start 
Program summarizes and reports on the program at the end of each project year (June). Staff 
prepare a report of their data, with numerous tables and charts, for their own and others’ use. 

The following paragraphs are drawn from the data supplied in staffs report (Even Start Staff, 
1999) and serve to describe the program, its families and their characteristics, and selected 
program accomplishments. 

In June of 1 996, 43 families in the Monongalia County Even Start Program participated 
in at least two of the program components (parenting education and early childhood education). 
The composition of families in the program was 27 (63%) couple households and 16 (37%) 
single households. Collectively, these families have 82 children participating in the program. 
The average number of children for all families was 2.3, but there was a difference across family 
type in that the average number of children in the couple households was 2.5, while the average 
in the single households was 1 .8 children. 
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Figure 1 . Graphic Depiction of the 
Monongaiia County Even Start Program 
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The distribution of the ages of the 82 children was rather even, ranging in number from 8 
to 15. There were more children in the 4- and 5- year old category, (14 and 15 respectively) 
followed by 1 1 and 10 in the 2- and 3- year old categories, respectively. There were 8 children in 
each of the 0-1-, 1-, 6-, and 7- year old categories. The number of 4-year-olds participating in the 
Even Start program decreased slightly from 19 in the 1996-1997 year to 14 in the 1997-1998 
project year. The percentage of those 4-year-olds enrolled in Head Start is viewed as one 
measure of this program’s success. Staff figures show a steady increase in the percentage of 4- 
year-olds enrolled in Head Start from 30% in 1993-1994, to 57% in 1994-1995, to 67% in 1995- 
1996, to 83% in 1996-1997. There was a slight decline to 71% in the 1997-1998 year. 

The average yearly income for all 43 families in the Monongalia County Program in 
1997-1998 was $1 1,257. However, this varied greatly by family type. For couple households in 
the program, the average family income was $13,160; for single households, the average family 
income was $7,592. The family incomes ranged from a low of $2,412 up to a high of $28,000. 

The 1999 report prepared by Even Start staff displayed data across the program years 
(Even Start Staff, 1999). For example, 217 children have benefitted by participation in Even 
Start and 1 1 ,23 1 books have been circulated to their homes over the years. Even Start is a long- 
term program for families; seven families have participated for 4 years, twenty-one families for 
3 years, and fourteen families for 2 years. The average hours the adults spend in adult education 
per year is 13.2. Sixty parents in the program have returned to an educational setting (17 earned 
a GED or EDP) and 63 parents attained a job from the training they received. Since the program 
began, 3,296 home visits have been made and 5,025 hours of instruction have been provided. 



Comprehensive Evaluation 

In the fall of 1996, program staff contracted with AEL, Inc. (formerly, Appalachia 
Educational Laboratory) in Charleston, West Virginia, to conduct an independent evaluation of 
the Even Start program during its fourth year of operation. This evaluation was more 
comprehensive than previous efforts and was designed to analyze client outcomes data collected 
by staff, and to collect new qualitative data about the program and its processes. The four 
objectives of AEL’s evaluation were (1) to identify the essential components of the program and 
the implementation variation of each component, (2) to evaluate the implementation of each of 
the program’s components, (3) to assess the adult and parenting education outcomes of the 
program’s Sample Study adults, and (4) to assess the early childhood education outcomes of the 
program’s Sample Study children. 

The Monongalia County project was one of the 57 sites randomly selected to be in the 
special Sample Study conducted nationally by Abt Associates. More detailed participation and 
outcome data were collected from parents and children in the Sample Study and also there were 
additional criteria to be included in the sample. This dramatically reduced the Monongalia 
County subsample to 16 families out of the total of 62 in the program in 1996. There was some 
attrition within the subsample. One parent (mother) and one child from each of the 16 families 
were targeted for the expanded data collection. 
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Data for the AEL evaluation were collected in a variety of ways, including program 
records, observation of events, face-to-face interviews, and seven participant outcome measures. 
For adults, the Comprehensive Adult Student Assessment System (CASAS) Life Skills test was 
used to assess reading and mathematics literacy (CASAS, 1997), while the Home Screening 
Questionnaire (HSQ) assessed parenting skills (Coons, Gay, Fandal, Ker, & Frankenburg, 1981). 
For each child, the Preschool Inventory (PSI) assessed school readiness skills (Abt Associates, 
1994) and the Preschool Language Scale-3 (PLS-3) measured language development 
(Zimmerman, Steiner, & Pond, 1992). The three PLS-3 scales were Auditory Comprehension, 
Expressive Communications, and Total Language. These seven outcome measures were 
administered three times over the two treatment years: pretest, posttest (end of first year), and 
follow up (end of second year). Statistical analyses of the outcome measures were t-tests for 
dependent means by the matched pairs technique. Effect sizes were computed for each t-test 
followed by items or points per treatment month gained (or lost). Graphic displays of each 
client’s score on each measure were designed and published in the final report (Meehan, 1997). 

Thirteen conclusions were drawn from the evaluation findings, some of which are 
summarized here. The completed Innovation Configuration Matrix of the Monongalia County 
Even Start Program is an effective communication device to describe and differentiate this 
particular program from others, while the depiction of the eight essential components in six 
concentric circles is an accurate and useful portrayal of the program (see Figure 1). It was 
concluded that the program had a large impact on all tliree adult outcome measures: reading 
literacy, mathematics literacy, and parenting skills. The largest effect sizes were for the first of 
the child outcome measures — school readiness. From these results, it was concluded that the 
Monongalia County Even Start Program had tremendous impact on children’s readiness for 
school outcomes — nearly double that expected for normal child growth. Regarding child 
language development, the program had some impact on each of the three PLS-3 scales, but not 
as much impact as on the school readiness measure. In the area of needed improvements, there 
was not enough service delivery nor outcome data on the fathers in the subsample families to 
analyze. Fathers have not been as involved in the Even Start program as they should be. 

Overall, AEL’s first evaluation (Meehan, 1997) concluded that the Monongalia County 
Even Start Program is a comprehensive, rural, home-based family literacy program that produced 
many positive impacts on the adults and children studied. It was concluded that the Monongalia 
County Program staff have designed and implemented an effective rural, home-based Even Start 
Program for clients. 



Parent Outcomes Evaluation 

A second evaluation of the Monongalia County Even Start Program was conducted by 
AEL staff in the 1997-98 program year. The main purposes of this evaluation were (1) to 
evaluate the impact of the program on a sample of families not in the previous study in terms of 
adult and parenting education outcomes, and (2) to evaluate the program’s original parenting 
education literacy materials. Four objectives were designed to address those two purposes; (1) 
to assess the adult education outcomes of a sample of the adults, (2) to assess the parenting 
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education outcomes of a sample of the parents, (3) to compare the outcome measures used in this 
evaluation with those used in the prior evaluation, and (4) to evaluate the program’s original 
parenting education literacy materials (Meehan, 1998). 

The evaluation design for the adults was the pretest-posttest administration of two paper 
and pencil instruments. The Adult Basic Learning Examination (ABLE) is a set of tests covering 
the basic areas of reading, mathematics, and language arts (Karlsen & Gardner, 1 986a, 1 986b, 
1986c). Four ABLE scales were used in this evaluation and the trained Even Start Family 
Educators administered and scored them. The Parenting Index (PI) instrument is a combination 
of three previously-developed scales related to parenting (J. Tucker, personal communications, 
August 14, 1998). The PI yields four scale scores: parenting efficacy, parental responsibility, 
internal locus of control, and self-esteem. The Family Educators administered and scored the PL 
Learner verification protocols were one-page instruments of seven to nine items — mostly 
constructed-response type — designed to assess the four original parenting education literacy 
materials. 

The number of adults with completed pretests and posttests of the two outcome measures 
varied; 13 for the ABLE instrument and 32 for the PI parenting education measure. The 
treatment period for ABLE spanned several years, while the treatment period for the parenting 
education measure was the 1 997-98 program year. A small, separate sample of parents 
completed the learner verification protocols and there was no connection between these measures 
and the two outcome measures. The pretest-posttest scores were analyzed with t-tests for 
dependent means by the matched pairs technique on each of the seven outcome scales. Next, 
effect sizes for the t-tests were computed and effect size descriptors were used to describe those 
numerical results. 

With respect to the literacy materials, an AEL evaluator analyzed the learner verification 
protocols completed by the parents. Each of the four literacy guides were evaluated separately 
and then a summary was prepared related to the multiple criteria of interesting, understandable, 
informative, utility, and attractiveness. 

The results were presented in tables and narrative copy (Meehan, 1998). Then, 
conclusions were drawn from those results, many of which are summarized here. Regarding the 
ABLE instruments, the Monongalia County Even Start Program had some impact on parents’ 
reading comprehension, a moderate impact on parents’ number operations, a very large impact on 
parents’ problem solving, and a large impact on their total mathematics scale score. Across the 
four ABLE scales, the Even Start program had much more impact in the mathematics area than 
in the reading comprehension. With respect to the PI instrument, it was concluded that the 
program had no impact on adults’ parenting efficacy, little impact on adults’ parental 
responsibility, some impact on parents’ internal locus of control, and some impact on parents’ 
self-esteem. From the field test of the four parenting education literacy materials it was 
concluded that they were very good in terms of being interesting, understandable,, informative, 
and having utility. However, it was concluded that the materials were poor in terms of their 
attractiveness, but that they could be improved rather easily. 
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With this second AEL evaluation completed, Even Start staff have evidence of positive 
outcomes of two different instruments for each of the two major areas of adult education and 
parenting education. There is no single, “best” instrument for either component. Deciding which 
instrument to use is a decision for staff to make after a careful discussion of relevant selection 
criteria (e.g., cost, ease of administration, relation to curriculum/services, etc.). 



Purpose of this Evaluation 



With two evaluations completed using two different instruments each to assess 
adult/parent outcomes and just one instrument (PLS-3) to assess child outcomes, the Monongalia 
County Even Start staff chose to explore other child outcome measures to administer during the 
1998-99 program year. After reviewing several alternative measures, staff selected the widely- 
known Peabody Picture Vocabulary Test-Ill (PPVT-III) as a new child outcome measure. Staff 
administered the PPVT-III to a sample of Even Start children on a pretest-posttest basis in the 
1998-99 year. 

The Even Start staff contracted with AEL staff to statistically analyze the PPVT-III scores 
from their administration. This was the third evaluation of the Even Start program by AEL staff. 
Too, it was much narrower in scope than either of the prior evaluations. The main purpose of 
this evaluation was to take the PPVT-III testing results sheets^compiled by the Even Start staff, 
create an SPSS database for the scores, analyze the data with appropriate statistical techniques, 
display and discuss the results, and draw conclusions from those results. The sole purpose of this 
evaluation effort was to analyze, display, and discuss the results of the Even Start staffs 
administration of the PPVT-III measure to a sample of Even Start Family children in the 1998-99 
program year. 



Audience for this Report 



The primary audience for this report is the Monongalia County Even Start Program staff 
and the host agency, the Monongalia County (WV) School District. This report is expected to be 
useful to staff to document the impact of the program on the vocabulary of the children in Even 
Start families, and to make decisions regarding the usefulness of the PPVT-III instrument in 
assessing child literacy outcomes. Secondary audiences include state administrators of the Even 
Start Program, administrators interested in evaluating child literacy program outcomes, and 
others interested in the impact of home-based, family literacy programs. 

The next section presents the methodology and the instrument employed in this 
evaluation. That section is followed by sections on findings, conclusions, and recommendations. 
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METHODOLOGY 

This section describes the methodology employed in this evaluation. First, the general 
design of the evaluation is presented, followed by a description of the Even Start children in the 
sample, followed next by an explanation of vocabulary instrument itself, and ending with the 
data analysis procedures. 



Evaluation Design 



This is an evaluation of the vocabulary outcomes of the Even Start program for the 
children in the client families. The language arts and readiness for school outcomes for a sample 
of children in the program were evaluated previously by AEL staff (Meehan, 1997). The 
Monongalia County Schools Even Start staff chose to expand the evaluation of the literacy 
outcomes for children by including a measure of vocabulary in the next program year. After 
reviewing several alternative measures. Even Start staff selected the Peabody Picture Vocabulary 
Test-HI to administer in the 1998-99 program year (Dunn & Dunn, 1997a, 1997b). 

To assess changes in vocabulaiy in the Even Start children, staff administered the PPVT- 
III on the pretest-posttest basis in the program year. Even Start Family Educators individually 
administered the PPVT-III in the fall as part of their planned home visits. Then, they 
administered a different form of the PPVT-III in the spring, again during their planned visits to 
the Even Start families. The Even Start Family Educators scored the completed PPVT-III 
instruments and compiled the results, along with test administration dates, on a summary sheet. 
This summary sheet included the child’s date of birth, test form, the raw score, and the standard 
score. This summary sheet used a project identification number for each child instead of actual 
names. Thus, AEL evaluators did not know the names of the children in the sample because 
AEL did not have a name-to-number listing. 



Children in Sample 



The number of families in the Monongalia County Even Start Program varied over the 
years as a result of a 1996-97 program self-assessment analysis. For example, in the program’s 
early years, there were up to 62 families enrolled. Beginning in 1997-98, 45 families were served 
so that the intensity of contact hours could be increased (Even Start Staff, 1998). Within each 
family, one adult and one child were targeted to complete the outcome measures. However, for 
some of the families, the targeted child was below the age for which PPVT-III norms applied. 

The sample of children completing the PPVT-III in the 1998-99 program year were those in 
families enrolled in the program for that year who were old enough to have norms in the PPVT- 
III technical manuals. The number of families enrolled declined slightly from the previous year. 



ERIC 
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Not all targeted children completed both the pretest and posttest forms of the PPVT-III. There 
were 32 children in the final sample with both pretest and posttest scores (one child invalidated 
the pretest administration). 

Within the sample of 32 Even Start children in this study were two subgroups. These 
subgroups were formed on the basis of additional information about them that the Even Start 
Family educators possessed. In one subgroup were children for which English was identified as 
their second language. Five children were in that ESL group. In the other subgroup were 
children who had an active individualized education plan (lEP). There were 10 children in that 
subgroup. The two subgroups were not mutually exclusive, but only one child in the sample was 
in both subgroups. 



Vocabulary Instrument 



The Peabody Picture Vocabulary Test-Ill is an individually administered, untimed, norm- 
referenced test. As the name indicates, the present version is the third one in the series. (The 
first two were published in 1959 and 1981, respectively.) The PPVT-III is designed for persons 
aged 21/2 through 90+ years. It serves two main purposes: (1) as an achievement test of receptive 
(hearing) vocabulary attainment for standard English, and (2) as a screening test of verbal ability. 

The PPVT-III is available in two parallel forms, designated as Form IIIA and Form IIIB. 
Each form contains four training items and 204 test items. These test items are grouped into 17 
sets of 12 items each. The items are arranged in order of increasing difficulty. Each item 
consists of four black and white illustrations arranged on a page called a picture plate. The task 
of the subject is to select the one picture on the page that best represents the meaning of the 
stimulus word presented orally by the test administrator. Item sets that are far too easy or too 
hard are not administered. Testing time averages only 1 1 to 12 minutes because most individuals 
take only five sets, or 60 items of the appropriate difficulty. 

Reliability information for the PPVT-III is presented in the teclmical manual (Chapter 4, 
Williams & Wang, 1997). The internal consistency reliabilities (Coefficient Alpha) for both 
forms of the PPVT-III ranged from .92 to .98 for the 25 standardization age groups with a median 
reliability of .95, also for both forms. The split-half reliabilities for the same 25 standardization 
age groups ranged from .86 to .97 for both forms, with a median reliability of .94 for both Form 
IIIA and Form IIIB. All persons in the standardization groups took both forms of the PPVT-III, 
allowing the computation of the alternate-forms coefficient of equivalence, a correlation. These 
correlations across the two forms ranged from .88 to .96 with a median of .94, indicating quite 
parallel forms. Test-retest reliability is an index of the stability of the test’s scores. Here, the 
developers randomly selected 226 persons in four of the age groups and administered both forms 
of the instrument with an average of 42 days between administrations. Test-retest reliabilities for 
the four age group samples ranged from .91 to .94 with little or no difference between the two 
forms. 
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Validity information for the PPVT-III is presented in Chapter 5 of the teclinical manual 
(Williams & Wang, 1997). In terms of content validity, items were selected from a pool of 
standard English words in 20 common content areas and that could be depicted by an illustration. 
Guidelines for selection included the following: related to important life skills; not to use 
homonyms, words that required specialized knowledge, or words adapted from another language. 
With respect to construct validity, the PPVT-III is a test of vocabulary; therefore, it is a measure 
of verbal ability. The developers cite data from several other tests that support the conclusion 
that the vocabulary section is the most important scale or test within those measures. But, the 
PPVT-III measures receptive vocabulary only, while the others measure expressive language. 

For criterion-related validity, the PPVT-III was correlated with three intelligence and two oral 
language measures. Correlations of both of the PPVT-III to the intelligence measures’ 
vocabulary sections ranged from .80 to .92. Correlations of both forms of the PPVT-III to the 
pair of oral language scales ranged from .63 to .83. Last, the PPVT-III did discriminate among 
special populations of children and adults who were matched with similar children and adults 
from the national standardization study. 



Data Analysis Procedures 



A unique and cost effective aspect of this study was that the majority of the data for this 
evaluation were collected by the Even Start program staff. As explained above, the Family 
Educators administered the PPVT-III twice to each child during the year (fall and spring). They 
scored the completed instruments and recorded the information on summary sheets. Using 
project-supplied identification numbers for each child, the summary sheets also included the 
child’s date of birth as well as the test form, raw score, and standard score for each 
administration. 

The PPVT-III summary sheets were mailed to AEL for analysis and interpretation. AEL 
evaluators created an SPSS database for the summary sheet information. AEL staff decided to 
add the PPVT-III age-equivalent score to the database for use in analysis. Actual ages in months 
were computed from the birthdate and test administration dates. The age-equivalent values were 
found in the PPVT-III technical manual (Williams & Wang, 1997). 

Analysis of the treatment months was straightforward. Instrument administration dates in 
the SPSS database were used to compare the treatment months. Here, months were rounded to 
the nearest whole month with the 15''' day as the midpoint (except February). These were done 
by hand and double checked for accuracy. 

Next, descriptive statistics of both the pretest and posttest PPVT-III scores were 
generated. The coefficient of variation (the standard deviation divided by the mean) was 
computed manually and double checked. These data were compiled into one table with each 
administration being one row of data. 
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Then, t-tests for dependent means by the matched pairs technique were computed by the 
SPSS software for the PPVT-III scores. This was done for the full group first, the active lEP 
subgroup second, and the ESL subgroup third. The coefficients of variation were computed next. 
Effect size, defined as “the degree to which the phenomenon is present in the population” 

(Cohen, 1977, p. 9) or an “indication for practical meaningfulness” (Fan, 1999), were also 
calculated. Cohen’s formula was used to compute the three effect sizes and his descriptors were 
employed. This was done by hand and double checked for accuracy. Then, a table was 
constructed of the essential t-test data, the coefficients of variation, and the effect sizes. 
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EVALUATION FINDINGS 



This section presents and displays the findings from the evaluation of vocabulary 
outcomes for the children in the Monongalia County Even Start Program. 



Ages and Treatment Months 

The average age of the 32 children in this study was 5 1 .67 months or four years and four 
months. The mode and median age in months both were 5 1 .00. 

The treatment period for this evaluation was the 1 998-99 Even Start program year. 
Pretests and posttests of the PPVT-III were completed as part of the regular home visits by the 
Even Start Family Educators. Of the 32 children with both scores, 15 had eight months of 
treatment, 10 had seven months, and the rest of the children had nine (N=l), six (N=2), five 
(N=2), or three (N=2) months. The mean number of treatment months was 6.93 with a standard 
deviation of 1.638. 



Vocabulary t-Test Results 

Table 1 displays the descriptive statistics for the PPVT-III pretest and posttest standard 
scores. Note that there was one pretest score for one child that was not usable. Possible standard 
scores could range from 90 to 160 points. The pretest scores ranged from 53.00 up to 1 15.00 
with the mode and median both being 93.00. The mean pretest score was 90.53 and the standard 
deviation was 14.37, yielding a coefficient of variations of .159. The standard eiTor of the mean 
was 2.54. 

At posttest administration of the PPVT-III to the Even Start children, the scores ranged 
from 77.00 to 1 19.00. The posttest mode increased five points to 98.00 while the median 
increased seven points to 100.00. The mean posttest score increased more than nine points to 
99.76 while the standard deviation decreased more than five points to 9.24. The coefficient of 
variation and standard error of the mean both declined to .093 and 1 .6 1 , respectively. 
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Descriptive Statistics for the PPVT-III Pretest and Posttest Standard Scores 



Number 

of 

Children 


Minimum^ 


Maximum 


Mode 


Median 


Mean 


Standard 

Deviation 


Co - 

Efficient 

of 

Variation 


Standard 
Error of 
Mean 


Pretest A dministration 


32 


53.00 


115.00 


93.00 


93.00 


90.53 


14.37 


.159 


2.54 


Posttest A dministration 


33 


77.00 


119.00 


98.00 


100.00 


99.76 


9.24 


.093 


1.61 



“ Standard scores could range from 40 to 160. 
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Table 2 displays the pretest and posttest PPVT-III standard score t-test and effect size 
results for the full group and two subgroups of Even Start children. For the 32 children with both 
scores there was a 9.28 point gain on PPVT-III vocabulary test. This gain yielded a t-value of 
4.51, which was significant at the .0001 level. The effect size for the gain in the vocabulary 
score was .65, which is “moderate” in Cohen’s set of descriptors. 

The t-test results for the 10 children with active lEPs are displayed in the second row in 
Table 2. For the active lEP children, the vocabulary gain was much larger than the full group: it 
was a gain of a full 15.50 points. The t-value of 3.56 was slightly less (owing to the smaller size 
of the subgroup), but still it was significant at the .01 level. The effect size was 1.13, above the 
start of the “large” category (.80) in Cohen’s scheme. Since it is more than the .30 difference 
between both the small to moderate and moderate to large category values, the descriptor of 
“very large” was chosen to describe the effect size for the active lEP subgroup on the vocabulary 
measure. 

The t-test results for the five children in the ESL subgroup are displayed in the third row 
in Table 2. Again, at 14.80 points, the ESL subgroup gain was much larger than that of the full 
group and just a little less than that of the active lEP subgroup. The t-value of 3.37 was slightly 
less than that of the active lEP group (owing, again, to the smaller size of the subgroup). 
However, the probability of .028 was significant at the .05 level. The ESL subgroup effect size 
was 1.05 , which is large in Cohen’s scheme. 



Age Equivalent Results 

In addition to the analysis of the standard scores, the age equivalent scores for the PPVT- 
III pretest and posttest administrations were analyzed. The average age-equivalent of vocabulary 
for the pretest administration, for the full group, was 3.713 or three years and seven months. The 
pretest range was from 1.1 to 9.1 with a median of 3. 075. The average age-equivalent of 
vocabulary for the posttest administration was 4.845 or four years and eight months. The posttest 
range again was from 1.1 to 9.1, but the standard deviation decreased from 2.105 to 1.852. The 
posttest median age-equivalent increased to 5.020. 




26 



